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NAVAL POSTGIiADUATE SCHOOD 
MONTE^Y, CALIF. 9394Q 



ABSTRACT 



This report describes a project which developed an objective 
measurement of motivation using Flanagan's critical incident 
technique. 'Hie sample population was U, S. Navy Jezebel operators. 
The report includes a rating form which was used, the statistical 
computations of the data recorded and a new, modified form. The 
rating form was found to have a high interrater reliability; many 
of the scores on the items had an exceptionally high correlation 
with the total score received and the study generated a particularly 
good operational definition of motivation. 
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I . INTRODUCTION 



The problem of increasing productivity or improving performance 
is one which concerns everyone in a managerial position. Numerous 
studies have been conducted to identify and attempt to control as 
many of the factors that have a bearing on output as possible. One 
of the best descriptions of the interrelated nature of the many 
elements involved can be found in Sutermeister (Sutermeister, 1969) . 
In his conceptual scheme the factors most closely related and 
directly influential on productivity are the physical elements of 
the job (i.e., technological development, raw materials, job layout 
and methods) and the employee's job performance which he attributes 
to a complex interaction of ability and motivation. While ability 
is not a precisely defined or completely understood concept it is 
measurable and reliably modifiable. The motivational level of 
employees is a different matter altogether, both in terms of the 
degree of agreement on the elements of motivation and in the 
untapped potential for greater understanding and selective modifi- 
cation of motivation. It can safely be said that there is virtually 
universal acceptance of the important influence that motivation has 
in a broad range of human endeavor. 

In the past 25 years there has been a proliferation of theories 
of motivation. Many of these, such as Maslow's "Need Hierarchy 
Theory," (Maslow, 1943) , "McGregor ' s Theory Y" (McGregor, 1960), 
Victor Vroom's "Instrumentality Theory" (Vroom, 1964) and Herzberg's 
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"Two Factor Theory" (Herzberg, 1959) have generated widespread 
interest. More recently several organizations have come into 
existence that purport to selectively modify the level of motivation 
of individuals, groups, or of entire organizations (for example. 
Success Motivation Institute of Waco, Texas) . 

There is one fundamental element that seems to be missing in 
both the theoretical and the applied approaches to motivation. This 
missing element is a method of determining the level of motivation 
(as represented in performance) of a group or an individual. Such 
a technique is necessary to enable investigators to detect differences 
or changes in motivation and to test the effects of incentives or of 
management policy. An intensive search of the available literature 
failed to locate an objective instrument for the measurement of 
motivation. 

It was important in the initial stages of this project to 
limit the area to be investigated and to select a definition of 
motivation upon which to base the study. It was assumed that 
motivation requires force and direction and the study was limited 
to the work context. Motivation then is task related effort expended 
in directed activity. In the investigation of the literature it 
became increasingly obvious that there are two quite distinct 
approaches to motivation, one is the classical psychological view 
of motivation (which refers to the processes governing choices made 
among alternative forms of voluntary activity) , the other is 
motivation as seen by the management theorists. This dichotomy seems 
to have developed from the method of approaching the problem. The 
psychologists have looked at physiological states, generally resulting 
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from deprivation, and have inferred motivation from observed 
behavior, content analysis, or stated attitudes whereas researchers 
dealing with an industrial or social organization have tended to 
view motivation as the element of performance that can not be 
explained by any other known variable. This investigation was 
oriented along the latter lines, that is the scope was limited to 
the world of work and the motivation to perform a particular task 
or group of tasks . 
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II. NATURE OF THE PROBLEM 



In the development of a measurement of motivation, the major 
problem was to determine what behavior is characteristic of a highly 
motivated task-oriented worker. Specifically it was necessary to 
determine whether most motivated men in a reasonably limited job 
category can be expected to be identifiable through a few acts or 
a few classes of characteristic action and whether certain actions 
are characteristic of all or at least a significant percentage of 
highly motivated people doing approximately the same job. The 
second important consideration was to insure that a reasonably 
accurate performance measure was available for the population 
sampled. Failure to consider this element would result in an 
instrument which could not be "calibrated" or tested in a real world 
setting. As a larger percentage of jobs have a less direct influence 
on production, this requirement becomes much more difficult to satisfy. 
Because of this, it was necessary to conduct the study outside the 
setting of the production line to insure relevance to modem industry 
and the Armed Forces . 

Whenever a measurement instrument purports to measure a concept 
as ill-defined, poorly understood and with the many dimensions of 
motivation, the problems of validity and objectivity become quite 
severe. Is it really motivation that we are measuring or some 
complex and obscure artifact? Do the raters see the behavior of 
the people they are rating as demonstrative of an element of a 
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complex characteristic? How competent are raters at generalization 
from one specific act to a global evaluation? How much is the 
evaluation of motivation influenced by performance? Indeed, how much 
can performance and motivation be separated? 

As a result of these questions it was imperative to select an 
experimental technique to minimize the influence of unwanted variables. 
A unique opportunity and certain necessary encouragement to study this 
problem were presented by Dr. John P. Smith of the United States Naval 
Personnel and Training Research Laboratory, San Diego, California. He 
and his Jezebel Study Group had developed an effective performance 
measure for operator proficiency and had done some work with a 
preliminary motivation rating scheme. As a part of the field testing 
of their proficiency test a set of general demographic information 
was gathered about each man. This included both biographical and 
experience data. As an element of this information, supervisors 
were asked to rate the men working for them on their degree of 
motivation. The instructions for this rating included the comment 
that "motivation implies consistent willingness to give effort to a 
task or duty even in the absence of demand by a military superior; it 
implies a belief in the importance of the Air ASW mission and it 
implies a constructive attitude toward other squadron personnel." 

The men were given a numerical rating from one to seven, this score 
was used for analysis and tests of correlation with the performance 
rating and the other factors. Their analysis revealed that two of 
the factors Crank and motivation) had a significantly higher 
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correlation with the performance rating than did the other elements 
and this contribution was not predicted by any of the aptitude or 
other personal characteristics such as GCT, ARI, age or education. 
The following paragraph is an excerpt from their report: 

TTie guidelines given to the raters to define motivation 
are consistent with characteristics which would be expected to 
be present in career personnel; if so, rank and motivation 
rating should be correlated. Investigation of this point 
resulted in a moderately low Pearsonian correlation coefficient 
of .36. The two variables are related, but they are also 
sufficiently independent of each other to strongly suggest 
that further study of this approach to motivation may be 
profitable. Further data analysis has been carried on to 
evaluate more precisely the significance of the motivation 
ratings using analysis of coveriance and multiple correlation. 

(Naval Personnel and Training Research Laboratory Technical Bulletin 

STB 11-5.) 
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III. EXPERIMENTAL PROCEDURE 



A. GENERAL PROCEDURE 

In the previous section a number of the problems inherent in 
developing an instrument for the measurement of motivation were 
discussed. A constant concern in the selection of an appropriate 
procedure and in processing the data that were generated was to 
minimize the influence of these problems. In keeping with this 
philosophy, the method of choice was John C. Flannigan's Critical 
Incident Technique. Very briefly the Critical Incident Technique 
involves describing a situation for the person being interviewed 
and asking him to relate an incident that fits this situation. 
Specifically, in this project the men were asked to, "Think of 
the most highly motivated Jezebel operator that you know. What is 
it about him that caused you to select him? Please be as specific 
as you can." After asking this question the only additional 
comments that were made by the interviewer were requests for greater 
amplification of the incidents that were reported or for more specific 
incidents (Flannigan, 1954) . 

Interviews were conducted on two occasions. The first set of 
interviews were with one senior petty officer from each of four VP 
(Patrol) squadrons and two petty officers from another squadron. 

Since this set represented a horizontal section across the rank 
structure the second interviews involved a vertical section of 
eight operators whose paygrade varied from E3 to E6 (AWAN-AWl) in 
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a single squadron. It was felt that this procedure would tend to 
minimize the possibility that a significant difference existed 
between the conception of motivated behavior held by junior and 
senior petty officers. The initial intent was to investigate the 
possibility of developing incidents descriptive of highly motivated 
and poorly motivated behavior. This was unsuccessful with the 
interview technique that was used; only positively motivated behavior 
was described by specific incidents. There are several possible 
reasons for this and they are described in the Results and Discussion 
Section. 

A total of 97 incidents were collected and after closely related 
items were combined or eliminated 55 separate items were retained 
for use. Several independent attempts were made to order these 
items into dimensions of motivation and to produce a scale of 
increasingly favorable or graded desirability. These attempts 
were unsuccessful; the incidents did not naturally fall into 
distinct categories and they did not have varying degrees of 
desirability. TVenty items were selected for inclusion on the 
rating form. The basis for selection was the frequency that a 
behavior was reported or the particularly characteristic nature 
of the behavior. It is important to note that in several interviews 
one incident was sufficient to cause a man to be singled out and 
described. In general the behavior that was described in this 
manner represented a considerable amount of time and effort on a 
project or a breakthrough that was achieved through intense interest 
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and attention to the airborne ASW problem. An example of the 
Motivation Rating Form is included as Appendix A. 

In March 1971, the form was tested at Naval Air Station, Moffett 
Field. The squadron (VP-19) had 18 Jezebel operators in the AW 
division and five senior petty officers familiar with each of the 
operators who were available to rate the men. The raters were 
asked to indicate when they were basing their evaluations on 
observation or an expectation. This was to attempt to separate 
actually observed behavior from an estimation that the man would 
reasonably be expected to perform in the described manner in an 
appropriate situation. After completing the form, each rater was 
asked to comment on his general attitude about the form, its 
acceptability, completeness and any specific comments about the 
description of motivation that it provided. Information from these 
interviews is the basis of part of the Analysis of Data section. 
Following the interview each rater was asked to rank order each of 
the men according to their mission-oriented motivation. They were 
asked to minimize the influence of the form they had just completed 
and to the best of their ability eliminate the influence of on the 
job performance. The information that was collected was analyzed 
using several statistical techniques that are described in the 
following section. 

B. SAMPLE POPULATION 

The population that was sampled in this project was drawn from 
United States Navy Jezebel Operators. Jezebel is a system for the 
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airborne classification and localization of submarines using 
information transmitted from sonobuoys . Specific details about the 
system and the techniques of analysis are classified, however, they 
involve a combination of training, experience, memory, mechanical 
and electronic devices and onboard reference publications. 

The minimum requirements to become an Antisubmarine Warfare 
Operator (AW), the rating which includes Jezebel operators, are a 
minimum General Classification Test (GCT) and arithmetic test (ARI) 
combination of 110, successful completion of an aviation physical 
examination and the applicant must volunteer for AW training 
(approximately 7% of the applicants are selected for training) . 

There are less than 2,100 AWs on active duty in the Navy and of these 
about half are designated Jezebel Operators.^ The training that 
leads to assignment to a flight crew in a squadron takes nearly nine 
months and is in the top 25% of Navy schools on the basis of the 
cost of training. After assignment to a squadron, training continues 
until designation as a member of an Alfa Crew which indicates achieve- 
ment of a high level of operational readiness and completion of a 

squadron training syllabus. Acquisition of basic skills, maintenance 
> 

of skill, and operational flying are a full time task. Because of 
this AWs are becoming more and more an autonomous group, freed from 



^This information was made available by 
rating detailer. 



the Bureau of Personnel 
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most other duties except watch standing and certain military 
obligations. They are not trained in a secondary technical 
specialty. 
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IV. PRESENTATION OF DATA 



In the instructions for the raters they were asked to circle 
the number that best fit the man's behavior in the area described 
by the item. They were further instructed to complete one item for 
all men, then to proceed to the next item and to complete it for all 
of the men. 

In preparing the data for analysis the score on the items was 
totaled to give an overall grade. In some of the analyses the 
total score was used and in other sections the score on a particular 
item or an average score was used. Averaging scores was possible 
since each man was rated by five independent raters . 

The scores that were recorded are included in the data 
section. 

A. INTERRATER RELIABILITY (IRR) 

The first analysis was a test for the interrater reliability 
of the form using the Pearson Product-Moment Correlation: 

_ NIXY - (ZXjClY) 

•^[NIX^ - (ZX)^][NZY'^ - (lY)"^] 

where: 

r is the correlation 
X is the value of one variable 
Y is the value of the other variable 
N is the number of cases 
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The results of this analysis are listed in Table I. The data from 
which the correlations were computed are included as Appendix C. 

The statistical significance was computed for each correlation 
using the ‘^Student’s t**: 



t = 



Cn-2) 

(1-r^) 



where: 

r is the correlation. 

n is the number of cases. 

The second statistical test was a comparison of the interrater 
reliability based on the rank ordering that the raters were asked 
to do after completing the form and the rank ordering generated from 
the scores taken from the rating forms. For this comparison the 
Spearman Rank Order Correlation rho (p) was computed. Both sets of 
correlations are included in Table II for ease of comparison. The 
formula for computing the Spearman rho is: 



N(N - 1) 

where: 

D is the difference in rank ordering for the two criteria 
being compared. 

N is the number of elements that have been ordered. 
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TABLE I 



Interrater Reliability 



Correlation^ ^ 

(between raters) n 



^ 1,2 


= 


.639* 


13 




— 


.545* 


13 


1,3 








" 1,4 


= 


.074 


13 






.617* 


14 


r 

1,5 












.824** 


13 


2,3 








To 1 


- 


.386 


13 


2,4 








""2,5 


= 


.931** 


14 




- 


.544* 


13 


, 

3,4 








r, r 


_ 


.916** 


14 


3,5 






r, r 


— 


.485* 


14 


4,5 







Note. — 

The subscript indicates the specific rater 
involved, i.e,, r^ 2 ^he correlation between the 

ratings of rater 1 and rater 2. 

^n = 14 whenever rater 5 is involved because he 
was the division CPO, familiar with each of the Jezebel 
Operators but not qualified himself and, as such, not 
included in the sample. In all other cases the two 
raters being compared were excluded from the sample. 

*Significant at the .05 level. 

**Significant at the .01 level. 
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TABLE II 



Interrater Comparison Table 



IRR IRR 

(Rank Ordering) (Form) n 



^1,2 


— 


.639** 


.761** 


13 


^1,3 


= 


.095 


.586* 


13 


Pi, 4 


= 


.100 


.01 


13 


^1,5 


= 


.266 


.746** 


14 


P2,3 


= 


.56* 


.745** 


13 


P2,4 


= 


0 


.195 


13 


P2,5 


= 


.604* 


.852** 


14 


P3,4 


= 


.123 


.255 


13 


^3,5 


= 


.600* 


.862** 


14 


P4,5 


= 


.139 


.313 


14 



*Significant at the .05 level. 

**Significant at the .01 level. 
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As can be seen from Colinnn 1 of Table II, there was only 
occasional agreement in rank ordering, however quite good 
agreement was achieved using the form. There was much better 
agreement among the raters when the form was used than when a 
global rank ordering was made. The reasons that the one rater, 
rater number four, tended to differ from the others can only be 
generalized from the situation and from other studies. (It is 
interesting to note that he was rated highly in motivation by his 
peers and he expressed very positive feelings about the form.) 
Perhaps the most likely explanation stems from the type of training 
that is done by the squadrons . After an operator completes the 
formal Navy school and is assigned to a squadron the remainder of 
his training is conducted by the senior operators in the squadron. 
When a man is assigned to a crew his sphere of contacts is reduced 
significantly. The result of all this (coupled with social inter- 
action) is an unequal familiarity among the operators. Many studies 
have been done on the reliability of human judges. The limitations 
and biasses are well known and include such phenomenon as "halo 
error," "leniency error," "similarity error," and others (see, for 
example, Kelly, 1967). The situation that exists in a typical 
squadron could easily amplify these errors and could distort or 
invert the relative rating given to some of the men by one of the 
raters. This is considered to be the most probable explanation 
for the one differing rater. 
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B. ITEM ANALYSIS 



The analysis of the responses to the individual items was 
undertaken using four general methods. First, the means and 
standard deviations for all responses to each item by each rater 
were computed, this was then broken down into two groups; one by 
the military rank of the person being rated and another according 
to the rater. Then a correlation of the average score on an item 
with the man's total score was calculated. Table III contains the 
composite score, the breakdown by seniority and the correlation with 
total score. Table IV is a listing of the means for each rater. 

Table III can be used to determine the items that contribute 
most to the total score and discriminate well among the men. If 
an item describes a degree of behavior that is observed or expected 
in all men then it should have a high mean and a low variance and 
will not have the ability to discriminate between men. Therefore, a 
mean near the midpoint is desirable as is a variance at least large 
enough to indicate a spread of scores. Another desirable character- 
istic for an item is a high correlation with total score. During the 
interviews conducted immediately after the forms were completed it 
became obvious that the importance of one dimension had been over- 
looked; that was the differences in behavior attributable to 
different levels of experience. As an example of this difference, 
a motivated man undergoing training as an AW might show his 
motivation by spending extra time in the flight simulators or 
trainers, whereas an operator who had achieved a high level of 
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TABLE III 



Item Comparison Table 



Item 


Mean 

All 


Var. 

All 


Mean 

Senior 


Var. 

Senior 


Mean 

Junior 


Var. 

Junior 


Corre- 

lation 

All 


A 


3.65 


1.74 


4.00 


1.53 


3.31 


1.75 


.832 


B 


4.19 


2.10 


4.97 


0.92 


3.41 


2.09 


.972 


C 


3.69 


1.72 


4.03 


1.24 


3.36 


2.03 


.737 


D 


4.27 


2.17 


5.05 


0.73 


3.49 


2.41 


.942 


E 


4.27 


2.49 


4.87 


1.80 


3.66 


2.50 


.832 


F 


4.29 


2.13 • 


5.00 


1.10 


3.59 


2.10 


.876 


G 


2.36 


4.21 


3.87 


2.38 


0.84 


1.45 


.920 


H 


4.09 


2.11 


4.67 


1.07 


3.51 


2.52 


.679 


I 


2.63 


3.80 


3.55 


2.31 


1.53 


3.42 


.923 


J 


3.29 


2.50 


4.17 


1.79 


2.38 


1.64 


.880 


K 


2.83 


4.17 


3.74 


2.50 


1.88 


4.24 


.869 


L 


2.60 


3.13 


2.72 


3.37 


2.49 


2.94 


.621 


M 


3.83 


2.61 


4.43 


1.86 


3.15 


2.63 


.921 


N 


3.99 


1.99 


4.50 


0.94 


3.46 


2.55 


.884 


0 


3.45 


2.40 


3.91 


1.26 


2.94 


3.22 


.780 


P 


4.05 


2.13 


4.67 


1.54 


3.39 


1.96 


.955 



Note. -- 



Correlation of the item average by all raters with total 
scores by all raters. 
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TABLE IV 



Rater Comparison Table 



Item 








Rater 






Difference 

Between 

Means 

(H-L) 


1 


2 


3 


4 


5 


A 


Mean 


3.40 


4.25 


3.73 


3.87 


3.00 


1.25 




Var. 


2.83 


1.80 


0.64 


1.05 


1.87 




B 


Mean 


4.13 


4.81 


3.73 


4.25 


4.00 


1.12 




Var. 


2.12 


1.76 


1.35 


2.47 


2.67 




C 


Mean 


4.80 


3.81 


3.47 


3.19 


3.25 


1.61 




Var. 


1.31 


1.63 


0.84 


2.16 


1.27 




D 


Mean 


5.20 


4.69 


4.27 


3.56 


3.69 


1.60 




Var. 


2.46 


1.69 


0.78 


2.26 


2.23 




E 


Mean 


4.93 


4.37 


5.20 


2.44 


4.56 


2.76 




Var. 


1.30 


2.12 


1.31 


1.99 


1.20 




F 


Mean 


3.67 


4.63 


5.00 


3.69 


4.50 


1.33 




Var. 


2.24 


2.92 


1.14 


2.10 


1.33 




G 


Mean 


2.80 


2.13 


2.07 


2.69 


2.13 


0.73 




Var. 


4.31 


4.52 


2.21 


4.50 


5.98 




H 


Mean 


4.67 


4.19 


3.20 


3.94 


4.44 


1.47 




Var. 


2.81 


2.96 


1.03 


1.80 


1.20 




I 


Mean 


4.60 


2.94 


2.43 


1.75 


1.44 


3.16 




Var. 


1.97 


3.53 


1.29 


1.13 


4.13 




J 


Mean 


2.87 


4.00 


3.00 


3.06 


3.31 


1.13 




Var. 


2.12 


2.27 


3.20 


2.20 


3.01 




K 


Mean 


5.13 


2.19 


2.67 


2.56 


1.63 


3.50 




Var. 


1.55 


1.63 


— 


2.86 


3.98 




L 


Mean 


3.80 


3.69 


2.47 


3.06 


0.06 


3.74 




Var. 


2.89 


1.56 


1.27 


0.73 


0.06 





(Table continued on next page) 
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TABLE IV (continued) 



Item 




1 


2 


Rater 

3 


4 


5 


Difference 

Between 

Means 

(H-L) 


M 


Mean 


3.60 


4.06 


3.71 


3.75 


3.94 


0.46 




Var. 


3.11 


3.53 


1.24 


3.00 


1.93 




N 


Mean 


4.67 


4.13 


3.75 


3.81 


3.50 


1.17 




Var. 


1.67 


2.92 


0.50 


2.70 


1.07 




0 


Mean 


4.8 


3.06 


3.75 


3.69 


2.25 


2.55 




Var. 


1.46 


1.26 


0.92 


2.76 


1.40 




P 


Mean 


4.46 


4.63 


4.42 


3.56 


3.31 


1.37 




Var. 


1.98 


1.58 


0.81 


2.40 


2.65 






Avg. Mean 


4.22 


3.85 


3.32 


3.30 


3.06 





competence would be unlikely to request any more trainer periods than 
the minimum monthly requirement. As a correlate, a motivated and 
experienced man might demonstrate his motivation by working to qualify 
in other crew positions, an activity that would not be expected of a 
junior man working toward his initial qualification. This differ- 
entiation by experience level existed in some areas, however it was 
not a characteristic of all items, many of the items were equally 
applicable to junior and experienced men. 
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The method used to evaluate items for their applicability to 
a particular experience group was to use the difference in the means 
of the two groups to identify items with a likelihood of having this 
characteristic. An item with a relatively high difference should 
favor the experienced group and a low difference should indicate an 
item of greater applicability to the lower experience level. (In no 
case was the mean of the senior operators lower than the mean of the 
junior operators. The average difference in the mean was 1.4 so that 
smaller differences tend to identify items more characteristic of 
junior men.) 

Another characteristic of a desirable item is that it should be 
marked in a similar range by all raters when rating the same group. 

Table IV shows the mean score on each item by each rater and the 
difference between the high and low score. A large difference tends to 
identify items that have been interpreted differently by different 
raters . An exception to this logic occurs when one rater has a mean 
which is much different from the others since either a high or low mean 
acts to limit the possible scores in one tail while increasing the 
possible scores in the other tail. In this case a slight change in 
the wording may greatly improve the suitability of the item. One 
rater, the rater whose interrater reliability was low, tended to differ 
markedly from the others at least twice as often as did any other rater. 

To illustrate how some of these desirable characteristics have been 
applied to the items, compare the figures in the last column of Table IV. 
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Three of the items have differences between means greater than 3.00 
(items I, K, and L) . On examination of these items it seems logical 
to assume that the interpretation could be different for the different 
raters. Probably item I is the item most susceptible to "extension 
into the future." That is, evaluating the phrase in the instructions, 
"if the circumstances were appropriate," to include situations 
considerably beyond that which was expected or interpreted by most 
raters . 

The wording of item K is such that it does not adequately describe 
or differentiate a highly motivated man. As it is written it could be 
broadly interpreted to include the majority of sailors and this 
apparently was the interpretation made by some of the raters. Item L 
was very much subject to the experience element described earlier. 
Limiting this item to evaluation for a particular experience group 
should ease the problem of interpretation. 

From this analysis a new rating form was developed and is included 
as Appendix B. This form should have greater interrater reliability, 
better discrimination between highly motivated junior men and the 
senior men whose motivation is marginal and it should be even more 
acceptable to the users. In preparing this form, items with a narrow 
variance or a very high or low mean were either modified or eliminated 
and the information from Table IV, together with a "face" determination 
was used to separate items by their applicability to a particular 
experience group. 
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C. POLICY CAPTURING 



Certain exploratory attempts were made to understand the criteria 
the raters used in the free rank-ordering. These efforts at policy 
capturing were rudimentary and not originally planned for inclusion 
in this study. The working concept was that in general raters tend 
to make evaluations based on a limited number of elements even if much 
more data is available (IVhisler § Harper, 1962). Because of this it 
seemed possible that the rank-ordering was made on the basis of only 
a few types of behavior and, if this was the case, they should be 
among the incidents described in the items on the form. To test for 
any primary relationship a scatter plot was drawn for each item showing 
the average score that the man was given by the five raters on the item, 
plotted against his average rank order. This was largely unsuccessful, 
although two items (0 and B) did have a plot somewhat similar to the 
expected shape. If a strong correlation had been found, it might have 
pointed to a simplification of the rating technique or of the definition 
of motivation. 

Ibis analysis v^as undertaken to determine if further, more exhaustive 
techniques such as factor analysis were warranted. Because of the weak 
correlation and the limited data no further investigation was undertaken 
at this time. 

D. PROBLEMS AND UNANSWERED QUESTIONS 

In the conduct of the project several problems came up which deserve 
discussion. First was the question of the character of motivation in 
the population sampled. Does the full spectrum exist from highly 
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favorable motivation to strongly negative motivation, or is the range 
limited (because of the selection procedures) to the area from neutral 
to highly favorable motivation? During the interviews it was impossible 
to generate neutral or negative incidents. It is difficult to 
conceptualize examples of low or negative motivation other than 
apathy, disinterest, or, in the extreme, destructiveness. This may 
be, in part, related to the questioning or interviewing technique 
used where positive or favorable situations were set up initially. 

If the procedure were reversed it is possible that neutral or negative 
incidents could have b.een developed. 

One problem that is frequently perplexing in the analysis of 
evaluations is the so-called "halo effect," the propensity for raters 
who know a man who should be rated high in one area (for example, 
promptness) to rate him high in other areas for which they do not 
have an adequate basis for evaluation. It can be expected that this 
would occur on a motivation evaluation. However, what may appear to 
be "halo error" is expected if each item is really measuring the same 
thing. Secondly, such an error may not be detrimental to the ultimate 
result in the early stages of development if certain necessary behavioral 
descriptions have not been included. This possibility should be 
alleviated when and if motivation is broken dov^m to dimensions which 
can be put on scales. IVhile motivation is very likely multidimensional 
the detailed investigation of these dimensions might best proceed from 
an initially global definition or treatment. 
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A clear and present problem in the measurement of any complex 
characteristic is the validity of the method chosen. This problem 
is a continuing one and one which can only be decreased with extensive 
and lengthy techniques. Further, it is intensified in the early stages 
of test development. There are at least two types of validity which 
the Critical Incident Techniques insure. First is what might be 
termed source validity; it is a valid measure of motivation because 
it was developed from descriptions of motivated behavior. To a 
degree this same fact can be considered to imply consenual validity 
and the consensus was further guaranteed by the post-test interviews. 
Face validity is at best an insufficient test, however, the form as 
administered certainly does have face validity. 

To improve the validity, to generate a numerical value for the 
validity of this instrument for measuring motivation will pose certain 
problems. First and most imposing is the problem that arises from the 
fact that this investigation was at the limit of current understanding 
of motivation. Because of this there are no other equal or similar 
intermediate criterion against which this can be compared. However, 
certain schemes can be envisioned which might involve setting some 
situation or series of situations in which a motivated man might be 
expected to perform in a particular fashion. These concepts are only 
fragmentary and conjectural and would require a great deal of additional 
thought and testing before they could be attempted. 
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V. CONCLUSIONS AND RECOMMENDATIONS 



The most significant conclusion that can be drawn from this study 
is that an instrument was developed that improved the reliability of 
ratings of a person's motivation. Since this work was in an area that 
has been virtually untouched, it was impossible to relate the results 
to any other findings for comparison. 

A second conclusion that seems to be warranted is that such an 
instrument would be applicable across a wide range of different skills 
and tasks with only minor modification. This conclusion was supported 
by the very general nature of the items that were generated in the 
interviews even though special attention was given to elicit and 
report incidents with the highest possible degree of specificity. 

This characteristic, while not completely unexpected, was encountered 
to a surprising extent. It is considered that a very few different 
forms could be developed to cover all but the most extraordinary of 
occupations or individuals. This conclusion will require additional 
study to verify. 

It is certainly not expected that a strong relationship exists 
between task or job motivation and career motivation. However, it does 
seem likely that a low task motivation should help to identify people 
who would be less desirable to retain in the military. 

There is a need to look at the variation of motivation over time 
and to test this variation with performance. The time dimension is 
one which has not been investigated. 
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The low relationship between morale or job satisfaction and 
production is well known, however, it seems possible that there may 
be a more complex interrelation with motivation being a key element. 
This possibility can be investigated through a motivation form. 

One of the most intriguing possibilities is the use of these 
findings and the methods that were used to discover and to 
appropriately value incentives. The relationship between the 
behavior that was reported as demonstrative of motivation and 
incentives that might be useful in motivating workers may at first 
seem remote. However, many times during the interviews the situation 
was an important element in the display of motivated behavior. It 
was often the case that the situation seemed to be the motivating 
factor and as such should be viewed as an important incentive. 

These descriptions were strongly supportive of Herzberg's "TVo 
Factor Theory," the most profound motivation came as a direct result 
of doing the job the men are trained to do. During the interviews 
it became obvious that problems in communicating ideas and value 
systems from the lowest levels in an organization to levels where 
meaningful action can be taken are still very much the rule. Attempts 
at uncovering valued incentives should help to bridge this gap. 

Most personnel evaluation systems require a rating of personal 
attributes or personality variables and the value of these variables 
to an organization is highly subjective at best. Personality traits 
are extensively used in evaluating the performance of people who are 
far removed from a direct influence on production. It may well be that 
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motivation is a trait that is meaningful to an organization and can 
be measured more precisely, reliably and more objectively than many 
of the traits currently being rated. There is the added possibility 
that it will have greater correlation with relevant performance than 
the other traits . 

The development of a well tested, reliable and accurate measurement 
technique for motivation and one that is relatively free from the 
effects of performance would provide a powerful new tool for further 
investigation of numerous concepts of management policy and leadership 
techniques. It is possible that changes in motivation might be a much 
more valid measure of leadership and management ability than output or 
performance in fields where ultimate criteria can only be inferred or 
that are strongly influenced by equipment differences or situational 
aspects which are not easily assessed. 

Certainly this project has done one thing and that is to generate 
an objective operational definition of task-oriented motivation. 
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APPENDIX A 



MOTIVATION 



Background Information : This evaluation form is an attempt to develop 

an objective measure of motivation. The items that are included have 
been taken from descriptions of highly motivated men. 

Directions : Circle the number that best indicates how characteristic 

the item is of the man being rated. The evaluation should be based 
upon direct observation wherever possible. IVhen direct observation 
has not occurred, indicate whether you think the man would act in the 
manner described if the circumstances were appropriate. This evalu- 
ation is directed at mission or job-oriented motivation only. 



Item 



Frequently Occasionally Never 
6 5 4 3 2 1 0 



A. Asks probing important questions; 
questions that are difficult to ans- 
wer and often require special effort 

or research. 654 321 0 



B. Doesn't wait to be told to do 
something, sees things that need 

to be done and does them. 654 321 0 

C. Seeks out other operators that 
you consider to be motivated, enjoys 
being around professionally competent 

men. Avoids poorly motivated men. 654 321 0 

D. Has taken time out to help some- 
one with a problem. 654 321 0 

E. Discusses advancement and profi- 
ciency exams with other operators at 
length shortly after taking them. 

F. Watches and assists mechanics 
or technicians in the repair of 
equipment to help them and to 
improve his own knowledge. 



6 5 4 3 2 1 0 



6 5 4 3 2 1 0 



33 



Item 



G. Has given lectures ans has pre- 
pared lesson plans when he was not 
required to do so. 

H. Has gone to extra effort to get 
information or to prepare himself 
ahead of time to help make a flight 
go smoothly. 

I. Has originated improvements in 
equipment or procedures . 

J. Has shown interest in the other 
crew positions in the aircraft 

and has taken positive steps to 
qualify in one or more, of them. 

K. Has come back to the 
squadron from TAD or even from 
leave to attend an important 
lecture or to complete an 
important task. 

L. Spends more than the 
minimum time in the trainers . 

Asks for additional trainer 
periods . 

M. Has contacted other 
squadrons, the Wing, the ASCAC 
or FAETUPAC to get any new or 
relevant information. 

N. Was reluctant or hesitant to 
leave an important or interesting 
situation directly related to the 
ASW mission. 

O. Starts the preflight checks of 
equipment prior to the normal time 
in order that any problems encoun- 
tered can be corrected. 

P. Has stayed on a job until it 
was finished regardless of regular 
working hours . 



Freq uently Occasionally Never 
6 5 4 3 2 1 0 

6 5 4 3 2 1 0 

6 5 4 3 2 1 0 

6 5 4 3 2 1 0 

6 5 4 3 2 1 0 

6 5 4 3 2 1 0 

6 5 4 3 2 1 0 

6 5 4 3 2 1 0 

6 5 4 3 2 1 0 

6 5 4 3 2 1 0 

6 5 4 3 2 1 0 
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The following items can be answered by either Yes or No. 



Item 



Q. Has learned a special procedure on his own 
from manuals and publications and is considered 
to be an expert in this area by the personnel in 
the squadron and, perhaps in other squadrons as 

well. Yes No 

R. Is interested in making sure that important 
intelligence information is sent to higher 

levels in the information chain. Yes No 

S. Has an eagerness to find and share new information 

with other squadron personnel. Yes No 

T. Assumes that the information that is being 
developed on a mission is important and will be used 
later on by someone who wasn't there when it was 

collected; keeps his logs and grams accordingly. Yes No 



Name of man being rated 
Rater's initials 
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APPENDIX B 



MOTIVATION 

(REVISED) 



Background Information : This evaluation form is an attempt to develop 

an objective measure of motivation. The items that are included have 
been taken from descriptions of highly motivated men. 

Directions : Circle the number that best indicates how characteristic 

the item is of the man being rated. The evaluation should be based 
upon direct observation wherever possible. When direct observation 
has not occurred, indicate whether you think the man would act in the 
manner described if the circumstances were appropriate. 

The first five items (A-E) should be completed for junior men only. 
Junior in this case should be considered as not having achieved Alfa- 
Crew status. 



Item 



Increasingly Favorable 



0 1 2 3 4 5 6 



A. Asks probing, important questions; 
questions that are difficult to answer 
and often require special effort or 
research . 



0 1 2 3 4 5 6 



B. Spends more than the minimum time 
in the trainers. Asks for additional 
trainer periods . 



0 1 2 3 4 5 6 



C. Seeks out other operators that 
you consider to be motivated, enjoys 
being around professionally competent 
men. 



0 1 2 3 4 5 6 



D. Starts the preflight checks of 
equipment prior to the normal time 
in order that any problems encoun- 
tered can be corrected. 



0 1 2 3 4 5 6 



E. Has gone to extra effort to get 
information or to prepare himself 
ahead of time to help make a flight 
go smoothly. 



0 1 2 3 4 5 6 
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TTie second five items (F-J) should be completed for senior operators. 



Item 



Increasingly Favorable 
0 1 2 3 4 5 6 



F. Has given lectures and has prepared 
lesson plans when he was not required to 

do so. 0123456 



G. Has come back to the squadron from 
TAD or even from leave to attend an 
important task. 

H. Has originated improvements in 
equipment or procedures . 

I. Has shown interest in the other 
crew positions in the aircraft and 
has taken positive steps to qualify 
in one or more of them. 



0 1 2 3 4 5 6 

0 1 2 3 4 5 6 



0 12 3 



4 5 6 



J. Has taken time out to help someone 
with a problem. 



0 1 2 3 4 5 6 



Tlie remaining items should be completed for all operators. 



K. Has contacted other squadrons, 
the Wing, the ASCAC or FAETUPAC to 

get any new or relevant information. 0123456 

L. Was reluctant or hesitant to 
leave an important or interesting 
situation directly related to the 

ASW mission. 0123456 



M. Discusses advancement and profi- 
ciency exams with other operators at 

length shortly after taking them. 0123456 

N. Has stayed on a job until it was 
finished regardless of regular working 

hours . 0 1 2 3 4 5 6 



0. Watches and assists mechanics or 
technicians in the repair of equip- 
ment to help them and to improve 

his own knowledge. 0123456 
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Item 



Increasingly Favorable 
0 1 2 3 4 5 6 



P. Doesn't wait to be told to do 
something, sees things that need to 
be done and does them. 



0 1 2 3 4 5 6 



Q. Is interested in making sure that 
important intelligence information is 
sent to higher levels in the information 
chain. 



0 1 2 3 4 5 6 



R. Has an eagerness to find and share 
new information with other squadron 
personnel . 



0 1 2 3 4 5 6 



S. Assumes that the information that 
is being developed on a mission is 
important and will be used later on 
by someone v^ho wasn't there when it 
was collected; keeps his logs and 

grams accordingly. 0123456 



Name of man being rated 
Rater's initials 
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Average 

Total 

Score 






Average Score on Item 






A 


B 


C 


D 


E 


F 


G 


H 


B 69.00 


4 


5.50 


3.20 


4.40 


4.50 


4.75 


3.75 


4.20 


39.75 


2.75 


3.30 


3.00 


2.40 


3.50 


4.25 


.50 


3.75 


22.80 


2.20 


2.00 


2.40 


1.80 


3.20 


1.60 


0.00 


1.40 


69.40 


4.00 


5.40 


4.80 


5.20 


5.40 


5.40 


4.20 


4.40 


54.00 


4.40 


4.00 


4.00 


4.40 


3.80 


4.00 


.80 


3.80 


F 54.40 


3.40 


4.40 


3.60 


4.20 


4.60 


4.40 


2.60 


4.00 


H 68.00 


3.60 


4.80 


4.40 


5.00 


5.40 


5.20 


4.20 


4.20 


K 50.60 


3.40 


4.20 


3.60 


3.40 


4.60 


4.40 


1.60 


4.20 


55.40 


4.80 


4.00 


4.60 


4.60 


4.50 


2.60 


2.60 


4.60 


K:: 54.20 


3.20 


3.60 


3.40 


4.40 


3.80 


4.20 


2.00 


4.40 


L 76.40 


4.60 


5.60 


3.60 


5.60 


5.40 


5.60 


5.20 


5.40 


M 50.00 


3.40 


4.00 


3.40 


4.00 


4.00 


3.80 


1.40 


3.80 


P 29.40 


2.40 


2.00 


2.20 


3.00 


2.20 


2.40 


.60 


2.60 


R 45.80 


3.00 


3.80 


3.40 


3.60 


3.60 


4.00 


.40 


3.80 


T 74.00 


5.20 


5.60 


4.20 


5.60 


4.20 


5.60 


5.40 


2.70 


W 67.20 


4.00 


5.00 


3.80 


5.00 


5.60 


5.40 


3.60 


4.60 



average 

Total 

Score 






Average Score on Item 






I 


J 


K 


L 


M 


N 


0 


P 


B, 


69.00 


3.00 


4.40 


3.00 


2.75 


4.50 


4.50 


3.50 


5.25 


^2 


39.75 


0.75 


2.00 


1.00 


2.00 


3.50 


3.75 


3.75 


2.67 


cl 


22.80 


0.00 


1.00 


0.75 


1.75 


1.00 


2.00 


1.50 


2.25 


c\ 


69.40 


3.25 


3.75 


4.80 


2.80 


4.80 


5.00 


4.50 


4.80 


cl 


54.00 


1.75 


3.25 


2.25 


3.00 


3.80 


4.20 


3.60 


4.40 


F^ 


54.40 


2.80 


4.00 


2.50 


1.60 


3.40 


3.60 


2.75 


3.60 


H 


68.00 


3.80 


3.75 


3.60 


2.20 


4.00 


4.40 


4.20 


5.20 




50.60 


2.25 


3.00 


2.00 


2.60 


3.00 


3.80 


3.00 


3.40 


^2 


55.40 


1.75 


2.25 


3.75 


4.20 


4.50 


5.25 


4.60 


4.20 




54.20 , 


2.20 


3.25 


2.50 


2.20 


3.75 


4.00 


3.25 


4.20 




76.40 


4.40 


5.20 


4.00 


3.60 


5.60 


4.60 


4.25 


5.60 


M 


50.00 


2.25 


4.25 


2.25 


2.60 


4.00 


3.80 


3.00 


3.80 


P 


29.40 


1.50 


2.00 


1.50 


1.60 


1.50 


2.25 


1.50 


2.25 


R 


45.80 


2.00 


2.25 


2.50 


2.40 


3.75 


3.75 


3.75 


3.40 


T 


74.00 


4.20 


5.25 


3.25 


3.20 


5.20 


5.40 


3.80 


5.20 


W 


67.20 


3.80 


4.00 


3.75 


3.20 


4.50 


4.75 


4.25 


4.20 



46 













. 4 




4 




BIBLIOGRAPHY 



Bernstein, L. L. A handbook o^ statistics solutions for the 
behavioral sciences . Holt, Rinehart, § Winston, I964~ 

Bruning, J, B,, § Kintz, B. L. Computational handbook of statistics . 
Scott, Foresman, 1968. 

Dunnette, M. D. Personnel selection and placement . Wadsworth, 1966. 



Dunnette, M. D. Managerial effectiveness: its definition and 

measurement. Studies in Personnel Psychology , 1970, 6-20. 

Flanagan, J. C. Principles and procedures in evaluating performance. 
Personnel , March 1952, 373-386. 

Flanagan, J. C. The critical incident technique. Psychological 
Bulletin , July 1954, 51 , 327-358. 

Fogh, L., Huhn, C. L., 8 Blood, M. R. Development of first-level 
behavioral job criteria. Journal of Applied Psychology, January 
1971, 3-8. 

Guilford, J. P. Fundamental statistics in psychology and education 
(4th ed.). McGraw-Hill, 1965. 

Herzberg, F., Mausner, B., 8 Syndurman, B. The motivation to work 
(2nd ed.). Wiley, 1959. 

Herzberg, F. Work and the nature of man. World, 1966. 



Kay, B. R. The use of critical incidents in a forced choice scale. 
Journal of Applied Psychology, April 1959, 43 , 269-270. 

Kelly, E. L. Assessment of human characteristics . Wadsworth, 1967. 

Maslow, A. H. A theory of human motivation. Psychological Review, 
1943, 370-396. 

McGregor, D. The human side of enterprise . McGraw-Hill, 1960. 

Murray, E. J. Motivation and emotion . Prentice-Hall, 1964. 

Naval Personnel 8 Training Research Laboratory, San Diego. Navv-wide 
standardized proficiency testing program for fleet Jezebel 
operators : III . Results . conclusions . and recommendations . 

March 1971, Technical Bulletin STB 11-5. 



47 



\ 



Procter, J. M. U. S. Naval Academy curriculum study ; III . Critical 
incidents . Washington, D. C.: U. S. Naval Personnel Research 

Field Activity, July 1959 (Bureau of Naval Personnel Technical 
Bulletin 59-21) . 

Reveal, R., Jr., § Ruch, F. L. An application of the critical- 
incident technique to Air Force combat leadership . Air Force 
Personnel and Training Research Center, February 1958. (Research 
Report AFDTRC-TN-53-8) 

Sutermeister, R. A. People and productivity . McGraw-Hill, 1969. 

Tyler, L. E. Tests and measurements. Prentice-Hall, 1964. 



Vallance, T. R., Glickman, A. S., ^ Vasilas, J. N. Critical incidents 
in junior officer duties aboard des troyer- type vessels . American 
Tnstitute for Research, April 1954 (Bureau of Naval Personnel 
Technical Bulletin 54-4). 

Vroom, V. Work and motivation. Wiley, 1964. 



Whisler, T. L. , 8 Harper, S. F. Performance appraisal . Holt, 
Rinehart, h Winston, 1962. 



48 



INITIAL DISTRIBUTION LIST 



No. Copies 

1. Defense Documentation Center 2 

Cameron Station 

Alexandria, Virginia 22314 

2. Library, Code 0212 2 

Naval Postgraduate School 

Monterey, California 93940 

3. Dr. William H. Githens (thesis advisor) 1 

Department of Business Administration 

§ Economics 

Naval Postgraduate School 
Monterey, California 93940 

4. Library 1 

Naval Personnel § Training Research Laboratory 

San Diego, California 92152 



49 



I 



IIMCIASSIFIED 



Spcuritv Classification 



DOCUMENT CONTROL DATA • R & D 

....sificat.on of tft/c. body of W ,nd».n.t anno, a, ion n.as, 6c cntc cd iW.cn ,6c ov,,a _ , / jy _ o,. 



|K originating ACTivtTY (Corporate author) 

Naval Postgraduate School 
Monterey, California 93940 



UNCLASSIFIED 



2b. GROUP 



I 3. REPORT Tl TL E 

AN OBJECTIVE MEASUREMENT OF MOTIVATION 



( A DESCRIPTIVE NOTES fTVpe of report and inclusive dales) 

Thesis 



[ 5 . AUTHORiS) (First name, middle initial, Jaat name) 

Ronald B. Pickett 
CDR, USN 



|6- REPORT DATE 




9. CONTRACT OR GRANT NO. 



b. PROJEC T NO 



96. OTHER REPORT N 0 ( S ) (Any numbers that may be assigned 

this report) 



10. DISTRIBUTION STATEMENT ^ J 1 -C 4- o 

This document has been approved for public release and sale; 
distribution is unlimited 



12. SPONSORING MILITARY ACTIVITY 

Naval Postgraduate School 
Monterey, California 93940 



I 13 ABSTRAC T 



This report describes a project which developed an objective measurement 
of »tivaur^strg Flanagan^s Incident f 

was U: S. Navy Jezebel operators. The report includes a rat g „,„^ified 
used, the statistical computations of the many 

form. The rating form was found to have a high ^^^errater reliabi y> J 



of 



the Scores on thi items had an exceptionally high 

score received and the study generated a particularly good operational 
definition of motivation. 



DD ,1473 



(PAGE 1) 



NO V 6 5 
0 / 02 - 016-6600 



50 



UNCLASSIFIED 



Security CUssi ficdtion 



Security Classification 



\ 4 

KEY WORDS 


LINK A 


link b 


L 1 K C 


ROLE 


W T 


ROLE 


W T 


ROLE 


W T 


Performance 

Evaluation 

Rating 

Critical Incident 
Jezebel Operators 















DD ."r.-UTS 'BACK, 



(PAGE 2) 



Security Classification 



TVjThesis 138485 

p4p 489 Pickett 

cic.2 An objective measure- 

ment of motivation. 



A Y 7(! 
9L ?: I 
Q MAP 79 
t FfO 64 
13 t/-! 84 



2 P 1 i» 7 
• 2 3 8 6 9 

2 5 b 1 

2 8 7 9 

r <r , V 



Thesis .1.38485 

P489 Pickett 

C.2 ,^n obiective measure- 

ment of motivation. 






thesP489 

An objective measurement of motivation. 




3 2768 001 92385 7 



